Report on the Dagstuhl Seminar 10412 QSTRLib: A Benchmark Problem Repository for Qualitative Spatial and Temporal Reasoning

Reasoning about spatial configurations, temporal constraints, or spatio-temporal dependencies is a major challenge in various application domains of current AI research. One of the research areas concerned with such topics is the field of Qualitative Spatial and Temporal Reasoning (QSTR). More precisely, research topics in QSTR are formalisms and algorithmic methods for processing qualitative information about continuous spatial and/or temporal aspects of physical reality. Thus QSTR forms a distinct subfield within the broader AI community working on knowledge representation and reasoning and has attracted research interest for more than 25 years now. One of the main ideas in QSTR is to describe spatial or temporal phenomena in the world in qualitative formalisms that are tailored to specific reasoning tasks. Qualitative abstraction allows for compact representations of infinite domains and hence is considered key to solve reasoning tasks on such domains efficiently. In view of this motivation of QSTR research, it seems indicated to evaluate the quality of both representation formalisms and reasoning techniques with respect to different criteria including but not limited to expressiveness, efficiency, and cognitive adequacy. In contrast to other established AI communities, the idea of benchmarking formalisms, reasoning procedures, and implemented systems has not played a prominent role so far. For example, performance analyses of QSTR algorithms or systems are often only conducted by empiric evaluations based on random samples of problem instances which in theory are known to be hard to solve. But this gives rise to the question whether evaluations on such problem instances is adequate, since random instances typically do not reflect problem structures that also occur in concrete problem domains. Moreover, little work has been done so far towards a systematic comparison of the variety of qualitative formalisms available in the literature, which seems a crucial deficit for practitioners who want to use QSTR formalisms in applications. The availability of a benchmarking repository is not only useful for system developers, benchmarking is also essential to evaluate new strands of research. This lesson can be learned from other communities in computer science, such as Automated Theorem Proving (ATP), Boolean Satisfiability (SAT), Constraint Satisfaction (CSP), and Automated Planning, where benchmark repositories have proven beneficial to estimate weaknesses or strengths of particular approaches and to identify reoccurring structural properties of problems or tasks. To initiate the development of a problem repository for QSTR, a AAAI Spring Symposium on Benchmarking of Qualitative Spatial and Temporal Reasoning Systems …